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This  research  is  part  of  an  ongoing  effort  to  understand  the 
processes  people  employ  in  reading  technical  material  and  the  ways 
in  which  computer  based  job  aids  can  assist  people  in  doing 
complex  tasks.  The  central  questions  were  ^Where  people  have 
easy  access  to  many  kinds  of  information  what  information  do  they 
select  to  help  them  do  the  job?  How  do  people  vary?  How  does  - 


MTieM  Of  I  NOV  IB  OBB^ETE 

«Al  01SS>kP4l44BBt 


SECURITY  CLABBIPICATIOM  OP  TNIB  PABI 


84  04  18  007 


ii^fomation  selection  differ-  across  sites  in  the  text 
end  graphics?^  This  report  provides  a  description  of 
the  final  work  in  this  area,  sponsored  by  our  contract. 


Accession  For 

KTIS  GRAftI 
DTIC  TAB 
UiifMwiouneed 
j-jsvtlf  Icatl  on_. 


1 


n 


Computer  Based  Job  Aiding: 
Problem  Solving  at  Work 


The  nature  of  work  In  many  occupations  Is  changing.  Routine  computation 
and  Information  gathering  are  becoming  less  critical  skills  for  workers.  No 
rows  of  men  In  eyeshades  and  cardboard  sleeve-protectors  laboriously  penning 
entries  Into  ledgers.  No  girls  pushing  carts  through  file  rooms.  Fewer 
mechanics  and  repairmen  poking  and  probing,  because  much  of  the  routine  diag¬ 
nosis  will  be  aided  by  computer-directed  diagnostic  routines  or  by  built-in 
diagnostics  In  complex  equipment.  And,  soon,  fewer  data  processors  feeding 
endless  raw  material  Into  computers— much  of  the  needed  Information  will  al¬ 
ready  be  stored,  and  more  new  data  will  be  entered  untouched  by  human  hands. 
There'll  be  less  need  for  people  to  talk  to  computers  because  more  computers 
will  talk  to  people. 


What  will  be  left?  Problems.  Problems  to  be  solved,  and  not  the  easy 
ones— they'll  be  handled  by  routine  programs.  But  there  will  always  be  prob¬ 
lems  that  Just  won't  fit,  problems  newly  Invented,  and  problems  whose  very 
nature  Is  undefined.  These  will  need  a  human  problem  solver.  The  critical 
skills  for  such  a  worker  will  not  be  to  gather  or  compute  Information  but 
to  select  Information  relevant  to  the  problem  and  to  manipulate  the  existing 
Information  structure,  to  turn  It  and  twist  It  to  bring  Into  focus  the  facet 
needed  for  this  problem.  More  work,  then,  will  Involve  human  problem  solving, 
aided  by  and  operating  on  computer  based  Information  structures. 


In  preparing  people  fbr  this  kind  of  work  and  preparing  Information 
structures  to  aid  them  at  work,  we  need  to  know  what  workers  need  to  know. 
Given  a  rich  Information  structure  and  a  problem  to  be  solved,  what  Infor¬ 
mation  do  people  select?  And  how  does  that  Information  selection  fit  Into 
the  larger  problem  solving  process?  This  paper  will  not  attempt  to  solve 
these  broad  problems  In  one  Jump,  but  will  make  a  start  by  defining  a 
context  within  which  to  view  the  problem,  describing  some  strategies  for 
studying  Information  selection  In  computerized  Job-aiding  systems,  and  pre¬ 
senting  Initial  findings. 

Studying  Problem  Solving 

In  studying  problem  solving  we  can  draw  upon  a  rich  body  of  literature, 
particularly  the  work  of  Newell  and  Simon  (1972)  and  associates.  This 
framework,  extended  and  refined  by  these  authors  and  others,  has  been  applied 
to  a  variety  of  content  areas.  There  has,  however,  been  little  examination 
of  the  on-line  problem  solving  strategies  of  Individuals  using  a  computer 
based  Job  aiding  procedure.  The  Newell  and  Simon  model  of  human  problem 
solving  seems  likely  to  be  useful  In  studying  these  processes,  but  although 
this  model  has  great  power  and  Is  potentially  applicable  to  a  wide  variety 
of  phenomena.  It  must  be  Instantiated  anew  In  each  new  setting. 


■  Mhlle  processes  such 

as  Input  translation  and  selection  of  a  problem  solving  method  probably  occur 
In  all  settings,  these  general  processes  or  others  may  appear  In  different 
forms  and  be  given  different  weightings  In  various  settings.  The  problem 
solving  processes  should  be  defined  within  a  specific  setting. 


Tht  sttting  to  be  considered  here  Is  a  computer  based  Job  aiding  pro¬ 
cedure.  Subjects  read  from  a  display  terminal  the  directions  for  a  procedural 
task,  construction  of  a  model  loading  cart.  It  was  possible  simply  to  read 
and  carry  but  each  of  the  steps  explained  In.  a  series  of  Instructional  (text) 
flrames.  If,  however,  the  subject  needed  more  Information  about  nearly  any 
term,  procedure,  or  graphic  display,  he/she  could  access  further  Information 
simply  by  "marking"  the  screen  with  a  light  pen. 

This  was  made  possible  by  analyzing  the  expected  Information  needs  and 
designing  brief,  readable  text  and  graphics,  likely  to  be  understood  by  most 
users  (Stone,  1977).  The  next  step,  though,  was  the  key— differentiating  this 
Job  aiding  procedure  from  many  others.  Including  previous  studies  In  this  re- 
search  program.  The  technical  Information  system  was  organized  as  Hypertext  , 
Incorporating  the  previously  available  surface  level  text  as  the  central  level 
In  a  three-level  hierarchy.  The  more  detailed  level  consisted  of  definitions 
of  terms;  some  definitions  were  verbal  only,  others  Included  both  text  and 
small  black  and  white  or  colored  graphics  of  parts,  their  spatial  orientation, 
or  operations  to  be  performed  on  the  parts.  The  more  global  level  here  con¬ 
sisted  of  graphics.  (In  other  cases  this  global  level  might  be  conveyed  verbally.) 

[Insert  Figures  1  and  2  about  here] 

The  three  levels  in  the  Information  structure  were  linked.  A  user 
operating  on  the  level  of  surface  text  (the  sequence  of  readable  but  suc¬ 
cinct  steps  In  assembling  the  loading  cart,  some  of  which  are  shown  In 
Figure  1)  could  touch  with  a  light  pen  almost  any  term  on  the  screen.  (See 
Figure  2).  The  program  would  branch  to  a  frame  defining  the  term,  and  the 
user  could  then  touch  the  word  Back  to  return  to  the  frame  In  the  surface 
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text.  Slallarly,  the  user  viewing  a  frame  In  the  surface  text  could  at 
any  point  touch  a  box  on  the  screen  In  order  to  view  a  graphic  Illustration 
of  the  object  (Figure  2).  From  a  frame  In  the  surfact  text  the  user  could 
move  to  graphics,  to  dictionary,  forward  to  the  next  frame,  or  backward  to 
view  a  previous  frame  of  the  surface  text. 

[Insert  Figure  3  about  here] 

Responses  to  Blocks  In  Information 

Even  when  Instructions  In  a  task  are  Informative  and  well  organized, 
an  Information  block  (such  as  failure  to  understand  the  referent  for  a 
technical  term  or  the  Intended  orientation  or  relationship  of  two  parts) 
may  occur.  What  responses  are  possible?  Ordinarily,  In  encountering  an 
Information  block  In  written  text  a  reader  has  only  a  few  options— to  plunge 
forward,  either  Ignoring  the  block  of  hoping  to  resolve  It  with  Information 
later  In  the  text,  or  to  move  backward  to  review  earlier  Information  that 
might  provide  clarification. 

Hith  the  use  of  Hypertext,  however,  the  reader's  potential  responses  are 
Increased.  Figure  4  represents  these  options  schematically.  Faced  with  a 
block  In  Information,  the  user  can  move  forward  (In  which  case  we  may  not 
know  a  block  has  occurred  except  through  performance  deficits  or  later  re¬ 
checks  to  that  frame),  move  backward  to  recheck  earlier  text  or  Hypertext, 
move  down  (an  arbitrary  designation  but  psychologically  useful  In  visualizing 
Information  moves)  to  check  the  definition  of  a  specific  technical  term  or 
even  the  meaning  of  a  familiar  word  In  this  context,  or  move  up  to  a  graphic 
view  of  the  assembly  completed  to  this  step  or  a  graphic  overview  of  the 
completed  object. 


[Insert  Figure  4  about  here] 


The  levels  of  the  hierarchy  available  as  responses  to  blocks  In 
.  InfbrMtlon  thus  Include  not  only  the  surface  text  of  directions  but  a 
■ore  detailed  level  and  a  nore  global  level.  This  conception  of  the 
hierarchical  structure  of  Information  Is  Implicitly  rooted  In  Frederlksen's 
(1975)  analysis  of  discourse  processes,  although  those  notions  are  opera¬ 
tionalized  In  a  new  way. 

Information  Search  In  Technical  Literacy  as  Problem  Solving 

The  dimensions  of  the  problem  to  be  solved  by  users  of  this  technical 
Information  system  can  best  be  understood  by  considering  the  several  levels 
of  goals,  the  problem  solving  processes  that  can  be  applied  In  this  context, 
and  the  means  by  which  these  processes  can  be  observed  or  Inferred. 

The  overall  goal  set  by  the  researchers  In  this  assembly  task  Is  to 
complete  the  correct  assembly  of  the  miniature  loading  cart.  Subordinate 
to  that  goal  are  the  subgoals  of  completing  each  of  the  sequentially  arranged 
text  frames,  each  guiding  transformation  of  the  object  from  one  state  to  the 
next.  (At  each  step  there  are  also  1  to  3  substeps  to  be  checked  off  by 
touching  the  light  pen  to  the  screen  as  completed,  but  our  analysis  focuses 
on  the  16  frames  for  surface  text;  the  17th  frame  requires  no  action.)  Sub¬ 
ordinate  to  completion  of  a  given  step  are  the  Information-seeking  moves  under¬ 
taken  within  that  step.  While  the  overall  goal  and  the  subgoals  are  the  same 
for  all  subjects  In  this  study,  the  Information  each  subject  requests  to  enable 
himself/herself  to  complete  each  step  may  vary  widely. 


The  focus  In  this  study  Is  on  the  problem  solving  strategies  Individuals 
use  In  overcoming  Information  blocks.  Little  emphasis  Is  on  the  evaluation 
and  selection  processes  that  guide  a  given  user  In  selection  of  Information 
from  the  Information  structure  provided,  from  his/her  own  Information  structure, 
and  from  the  object  In  hand;  emphasis  Is  on  the  directly  observable  moves  to 
various  parts  of  the  computer  based  system. 

Newell  and  Simon  (1972)  p.  88)  described  aspects  of  the  overall  organi¬ 
zation  of  an  Information  processing  system  In  this  way: 

1.  Input  translation,  producing  In  the  problem  solver  an  Internal  rep¬ 
resentation  of  the  problem  (In  this  case,  the  Information  block)  to  be  solved. 

"The  problem  solving  then  proceeds  In  the  framework  of  the  Internal  representa¬ 
tion  thus  produced  —  a  representation  that  may  render  problem  solutions  obvious, 
obscure,  and  perhaps  unattainable." 

2.  Selection  of  a  problem  solving  method:  "A  method  Is  a  process  that 
bears  some  rational  relation  to  attaining  a  problem  solution,  as  formulated 
and  seen  In  terms  of  the  Internal  representation."  In  this  case  the  avail¬ 
able  methods  may  Include  searching  the  total  Information  structure  (prior 
knowledge,  surface  text.  Hypertext,  and  the  object  Itself),  drawing  Inferences, 
self-monitoring,  and  Integrating  the  Information  bearing  upon  comprehension 

of  a  particular  point. 

3.  Application  of  the  problem  solving  method  to  the  Information  block. 

"At  any  moment,  as  the  outcome  either  of  processes  Incorporated  In  the  method 
Itself  or  of  more  general  processes  that  monitor  Its  application,  the  execution 
of  the  method  may  be  halted."  The  Information  block  may  be  overcome,  apparently 
overcome,  or  left  unresolved. 
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4.  Regrouping  If  necessary.  Newell  and  Simon  note  (p.  88,  footnote) 
that  "The  continuous  flux  of  new  information  from  the  environment  may  offer 
new  solution  possibilities  or  demands  that  cause  the  problem  solver  to  inter¬ 
rupt  (his/her)  current  activities  to  try  different  ones.”  They  also  state 
that  "when  a  method  is  terminated,  three  options  are  open  to  the  problem 
solver:  (a)  another  method  may  be  attempted,  (b)  a  different  internal  repre¬ 
sentation  may  be  selected  and  the  problem  reformulated,  or  (c)  the  attempt 

to  solve  the probi em may  be  abandoned,”  at  least  for  the  moment. 

( 

5.  Generation  of  subgoals.  While  the  method  is  being  applied,  new 
problems  may  be  recognized.  "The  problem  solver  may  elect  to  attempt  one 
of  these"  or  may  set  aside  new  subgoals,  "continuing  instead  with  another 
branch  of  the  original  method." 

In  order  to  understand  the  problem  solving  processes  engaged  in  during 
.computer-assisted  technical  literacy,  information  seeking  strategies  employed 
during  an  assembly  task  were  analyzed  in  terms  of  overall  group  frequencies 
and  individual  patterns.  The  questions  to  be  answered  were: 

1.  Is  performance  (with  Hypertext  availabel)  greater  than  in  previous 
studies? 

2.  What  is  the  relative  frequency  of  various  types  of  information  requests 
across  the  group? 

3.  How  do  programs  differ  in  the  relative  frequency  of  various  types  of 
information  requests? 

4.  How  do  individuals  vary  in  information  selection  and  use? 

a.  in  relative  frequency  of  various  information  requests 

b.  in  patterning  of  information  requests 


Method 

S—Dle 

The  subjects  In  the  sample  Included  20  adults  ranging  widely  In  back¬ 
ground,  Including  electrical  engineers,  stockboys,  secretaries,  graduate 
scents  and  teachers.  This  allowed  a  test  of  the  Hypertext  system's 
ability  to  respond  to  the  Information  needs  of  users  ranging  widely  In 
background. 

Materials  and  Equipment 

The  materials  used  Include  17  frames  of  directions  for  assembling  a 
miniature  loading  cart  from  Fishertechnick  (Leg-like)  blocks.  This  surface  text 
presented  each  step  clearly  and  concisely, as  shown  In  Figure  1.  (The  full 
text  and  all  segmented  graphics  are  available  In  Stone  &  Crandell,  1982.) 

Each  text  frame,  however,  gave  the  viewer  the  option  of  rechecking  text 
seen  earlier  or  branching  Into  Hypertext.  By  touching  with  a  light  pen 
almost  any  substance  worked  on  the  screen,  the  viewer  called  up  a  frame  that 
defined  the  term  or  explained  Its  meaning  In  that  context;  often  the  defini¬ 
tions  were  accompanied  by  small  Illustrations  of  the  part  or  operation  defined. 

The  viewer  also  had  the  option  of  viewing  several  types  of  graphics. 

By  touching  one  of  the  boxes  centered  below  the  directions  for  each  substep, 
the  user  could  view  an  Illustration  of  the  loading  cart  as  It  should  appear 
at  that  step  In  the  assembly  process.  Or,  by  touching  a  box  In  the  upper 
left  comer,  the  user  could,  at  any  point  In  the  program,  branch  to  a  graphic 
depicting  the  completed  object. 
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The  detached  display  screen  was  linked  to  Hazeltine's  TICCIT  (Time- 

shared  Interactive  Computer  Controlled  Information  Television).  A  central 

TM 

feature  of  the  system  for  this  purpose  was  the  Hypertext  Display  System. 
.Technically,  this  term  Includes  the  organization  of  both  surface  text  and  the 
supplemental  graphics  and  definitions,  though  we  will  at  times  speak  of  con¬ 
trasts  between  text  and  Hypertext.  Fuller  description  of  this  system  Is  given 
In  Stone  and  McMInn  (1982);  and  In  Stone,  Israelite,  Mudrick,  and  Hutson, (1983). 
Data-gathering  Procedures 

Each  subject,  tested  Individually,  was  seated  at  a  computer  display 
console  equipped  with  a  light  pen.  The  parts  needed  to  assemble  the  miniature 
loading  cart  were  spread  out  on  the  table.  The  examiner  explained  the  purpose 
of  the  study  and  led  the  subject  through  the  Introductory  phase  of  the  program, 
which  explained  and  Illustrated  the  Information  search  options  available.  Then, 
as  the  subject  progressed  through  the  steps  of  the  program,  the  examiner  made 
notes  on  his/her  performance.  An  Internal  tracking  program  recorded  and  later 
printed  out  the  sequence  of  numbers  of  the  frames  viewed  by  each  subject. 

Data  Analysis  Procedures 

The  key  variables  tabulated  were  number  of  requests  for  graphics 
(overview  or  segmented)  definitions,  or  rechecks.  Based  on  the 
conceptualization  of  Information  blocks  and  Information  search  strategies 
discussed  earlier,  the  computer's  Internal  tracking  records  of  the  sequence 
of  frame  numbers  viewed  were  transformed  (manually  In  this  case,  though  the 
TICCIT  system  could  be  programmed  for  this  task)  Into  visually  salient 
graphic  profiles  of  each  subjects'  performance.  This  facilitated  explicit 
description  of  each  move  by  each  subject,  yet  made  It  easy  to  detect  the 
larger  patterns  within  which  Individual  moves  were  embedded.  The  next 
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step  Mas  to  tabulate  relative  frequency  of  various  Infbrmatlon  requests  for 
each  subject  at  each  program  frame. 

Results  and  Discussion 

First  we  will  examine  performance,  then  the  overall  distribution  of 
various  types  of  Information  requests  across  all  subjects  and  all  program 
frames.  Next  we  will  break  down  the  distribution  of  requests  by  program 
frame  (reflecting  type  of  Information  block),  and  then  by  Individual  subjects. 
Individual  differences  will  be  examined  at  two  additional  levels,  the  overall 
pattern  of  Information  seeking  moves  by  selected  Individuals  and  the  problem 
solving  strategies  that  seem  to  be  reflected  In  these  patterns. 

Success  on  Performance  of  the  Assembly  Task 

One  variable  on  which  subjects  hardly  differed  was  success  on  the  task. 
The  directions  were  designed  to  be  clear  and  readable,  yet  previous  studies 
using  the  same  surface  text  (Stone  S  Glock,  1981)  showed  that  two-thirds  of 
the  subjects  completed  the  task  with  some  uncorrected  errors.  In  this  study, 
with  the  addition  of  Hypertext,  almost  no  errors  were  uncorrected  by  the  end 
of  the  task.  Nearly  all  subjects  succeeded,  but  they  succeeded  In  different 


The  technical  Information  system  made  possible  by  the  Hypertext  Display 
feature  embedded  In  the  TICCIT  system  In  effect  provided  for  each  Individual 
a  different  program  tailored  to  his/her  unique  preferences.  One  person  was 
able  to  view  many  supplemental  frames  while  another  saw  none;  one  to  view 
more  definitions  or  graphics  than  another.  This  carries  adaptive  Instruction 
to  great  lengths  and  Is  a  promising  approach  to  apply  when  users  are  potentially 
diverse.  An  expert  using  such  a  system  will  encounter  little  clutter,  while  a 
novice  finds  multiple  support  systems,  some  well  matched  to  his/her  preferences 


for  the  form  and  content  of  Information. 
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Overall  Frequency  of  Requests  for  Various  Information  Sources 

After  completing  the  assembly  tasks  In  frames  1  -  16,  subjects  found  In 
frame  17  an  expression  of  thanks  for  their  participation  and  an  Invitation 
to  explore  the  system,  rechecking  text  and  Hypertext  If  they  wished.  Some 
of  the  subjects  exercised  this  option,  most  for  Just  a  few  moves  and  others 
for  an  extended  search.  An  earlier  report  (Stone,  Hutson  &  Fortune,  1983) 
based  on  a  subset  of  this  sample.  Included  In  analysis  the  Information  requests 
during  both  assembly  and  exploration  phases.  Requests  for  graphics  and 
definitions  during  exploration  were  proportional  to  the  requests  during  the 
assembly,  though  the  percentage  of  rechecks  was  almost  by  definition  greater 
during  exploration.  The  present  analysis  will  Include  only  the  Information 
searches  during  assembly. 

[Insert  Table  1  and  Figure  5  about  here] 

The  surface  text  frames  were  viewed  by  all  subjects.  As  Indicated  In 
Table  1  and  Figure  5,  apart  from  surface  text,  47X  of  all  requests  were  for 
verbal  Information,  for  graphic.  The  verbal  requests  Included  definitions 
(37X)  and  rechecks  of  previous  text  (10%).  The  graphic  requests  Included 
segmented  graphics  (44%),  and  overview  graphics  (9%).  On  the  whole,  while 
there  were  more  requests  for  graphic  Information  than  for  verbal  Information 
(beyond  the  surface  text  viewed  by  all  subjects),  roughly  the  same  proportions 
of  verbal  and  graphic  requests  were  maintained  across  most  frames. 


Some  definitions  were  used  much  more  frequently  than  others.  It  would 
certainly  be  possible  to  reduce  the  lexicon  available,  but  a  definition  (either 
the  meaning  or  the  specific  referent)  needed  by  no  one  else  may  be  critical 
to  success  by  a  few  Individuals.  This  may  be  particularly  true  for  men  or 
women  with  little  previous  technical  experience  or  those  for  whom  English  Is 


a  second  language  or  for  fbreign  nationals  who  are  receiving  technical 
training.  In  a  new  study  of  comnunlty  college  students,  about  one-third 
were  speakers  of  English  as  a  second  language;  we  will  examine  closely  the 
kinds  of  terms  for  which  they  requested  definitions. 

Prevalence  of  Various  Types  of  Information  Requests 

Even  when  a  rich  technical  Information  system  Is  provided,  the  question 
might  be  raised,  “But  Is  all  that  necessary?"  The  second  aspect  of  analysis 
of  overall  Information  selection  addressed  that  question,  examining  how  wide¬ 
spread  was  use  of  a  given  type  of  Information:  Old  overall  means  result  from 
many  requests  by  a  few  Individuals,  or  did  many  subjects  request  a  given  type 
of  Information  at  least  once?  The  answer  seems  to  be  "Both."  While  some 
Individuals,  as  discussed  later,  made  much  higher  use  of  some  Information 
sources  than  did  others,  almost  all  Information  sources  were  used  at  least  once 
by  a  substantial  proportion  of  subjects.  The  surface  text  was  used  by  all  of 
the  subjects.  (In  this  case  they  had  to  check  one  or  more  steps  completed 
on  each  frame  In  order  to  move  to  the  next  surface  text  frame.)  Nearly  as 
many  subjects  requested  segmented  graphics  and  definitions  at  least  once. 

The  majority  requested  overview  graphics  and  rechecks  at  least  once. 

Did  they  need  all  these  types  of  Information?  We  can  only  answer  that 
the  subjects  apparently  felt  that  such  Information  would  be  helpful  In  solving 
a  probl em  encountered  during  the  assembly  task.  It  would  be  possible  to  reduce 
the  number  of  types  of  Information  available,  but  at  the  potential  cost  of 
reducing  adaptability  of  the  Job  aiding  procedure  to  a  wide  range  of  Individual 
d1 fferences . 


,  requests  for  definitions,  4  requests  for  overview  graphics,  and  10  rechecks 
of  text. 

(Insert  Table  3  and  Figure  7  about  here) 

Frame  5  provoked  more  text  rechecks  than  any  other,  as  though  the  new 
step  made  users  realize  that  there  was  something  earlier  they  hadn't  understood 
(Table  3).  The  two  clusters  of  frames  from  which  greatest  number  of  rechecks 
started  were  4-5  and  frame  10.  The  patterns  of  rechecks  In  Figure  7  give 
a  closer  look  at  rechecks  as  Information  search  strategies  and  the  sites  that 
provoke  them.  In  this  display  the  arrows  originate  In  the  frame  where  users 

felt  the  need  for  clarification  and  the  arrow  heads  Indicate  the  frames  to 

which  they  returned  for  clarification.  (In  longer  sweeps  they  may  also  have 
gained  some  Information  from  the  frames  viewed  enroute  to  the  terminus  of 
the  recheck,  but  apparently  not  enough  to  stop  the  search.)  Four  of  the  subjects 
made  no  rechecks,  while  some  made  extensive  use  of  this  option.  Many  of  the 
rechecks  made  were  brief,  such  as  subject  1's  three  separate  one-step  rechecks, 
but  a  few  were  longer,  one  recheck  covering  nine  frames.  The  rechecks  were 
predominately  for  surface  text,  but  Included  here  (though  not  double-counted  In 
analysis)  are  a  few  excursions  to  de'flnltlons  (shown  with  a  tick  below  the 
arrow  )  or  to  graphics  (shown  with  a  tick  above  the  arrow). 

The  largest  number  of  rechecks  (Table  3)  began  at  frame  5,  signalling  a 

problem,  and  the  largest  number  terminated  at  frame  4,  suggesting  that  In¬ 

formation  In  this  frame  helped  to  solve  a  problem.  Though  many  of  the  rechecks 
at  frame  5  terminated  at  frame  4,  there  was  not  a  point-to-point  correspondence. 
Some  of  the  rechecks  from  frame  5  passed  through  frame  4  and  continued  In 
reverse  to  earlier  frames,  and  other  rechecks  begun  at  frames  7  and  10  termina¬ 
ted  at  frame  4.  For  other  pairs  of  frames,  though,  there  were  apparent  links— 
rechecks  beginning  at  frame  3  all  ended  at  frame  2,  rechecks  beginning  at 

frame  8  ended  at  frame  7,  and  rechecks  from  frame  13  ended  at  frame  12.  Most 
of  the  rechecks  that  Involved  requests  for  graphics  or  definitions  centered 

on  frames  4  and  5,  and  a  substantial  proportion  of  these  were  by  the  same 

14 


IndlvIdiMl.  Both  the  smII  nunber  of  text  rechecks  and  the  uneven  proportion 
that  were  repeats  by  the  sane  user  suggest  that  while  this  Is  a  dimension  useful 
In  describing  Individual  differences,  use  of  text  rechecks  on  a  given  step  as  a 
signal  of  program  problems  would  require  a  larger  sample.  Once  a  problem  spot  has 
been  Identified  by  other  means,  though,  analysis  of  recheck  patterns  may  guide  flx- 
•up  strategies. 

Individual  Differences  In  Frequency  of  Various  Information  Requests 

Individuals  differed  widely  In  the  apparent  difficulty  (as  Indexed  by  number 
of  Information  requests)  they  experienced  across  the  whole  task  and  at  specific 
program  sites,  and  In  the  types  of  Information  they  requested. 

One  source  of  difference  was  In  relative  dependence  upon  graphic  versus  verbal 
Information  to  supplement  the  surface  text.  One  person  requested  graphic  Informa¬ 
tion  19  more  times  than  he/she  requested  verbal  Information.  The  most  verbally 
oriented  person  had  a  ratio  of  3  graphic  requests  to  33  verbal .  One  person  asked 
for  neither  type  of  Information. 

There  were  also  differences  In  Individuals*  use  of  specific  categories.  Several 
people  made  15-18  requests  for  segmented  graphics,  but  Just  as  many  made  only  2  or 
3  such  requests  and  one  person  made  none.  The  number  of  definitions  requested  by 
an  Individual  ranged  from  0  to  30,  and  the  total  number  of  Information  requests 
ranged  from  0  to  46. 

Reconstruction  of  Individual  Information  Search  Patterns 

Several  Individual  profiles  are  schematically  represented  In  Figure  8.  Movements 
from  left  to  right  represent  forward  moves  through  the  17  frames  of  the  surface 
text,  checking  off  various  steps  completed.  In  order  to  gain  access  to  the  next  frame. 
Requests  for  graphics  are  represented  as  movements  upward.  Requests  for  dictionary 
are  represented  as  movements  downward.  Rechecks,  requests  to  review  previous  frames, 
art  represented  as  movements  from  right  to  left. 
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Given  the  TICCIT  system's  Internal  tracking  programs.  It  Is  possible  to 
reconstruct  the  exact  sequence  of  Information  moves  made  by  an  Individual  and, 
within  limits,  to  Infer  some  of  the  problems  encountered  and  solutions  attempted. 

Subjects  varied  widely  In  the  number,  kinds,  and  patterns  of  Information 
requested.  Subject  7,  fbr  example,  made  no  requests  for  any  supplemental  In¬ 
formation  during  the  assembly  phase,  though  he  requested  two  graphics  during 
the  later  exploratory  phase  (not  Included  In  the  analyses  In  this  report). 
Subjects  10  and  14  requested  a  number  of  graphics  but  only  one  definition. 

Subject  8  had  a  balance  of  graphics  and  definitions. 

While  none  of  the  subjects  shown  In  Figure  8  made  rechecks  during  assembly, 
all  of  the  subjects  shown  In  Figure  9  made  some  use  of  rechecks.  During  the 
assembly  phase,  though,  their  rechecks  were  limited  to  the  Immediately  prior 
frame.  In  most  cases  the  rechecks  were  to  text,  but  for  subject  12,  who  made 
extensive  use  of  definitions,  one  recheck  was  also  for  a  definition,  and  for 
subject  4,  who  made  extensive  use  of  graphics,  the  two  rechecks  were  also  for 
graphics.  The  subjects  shown  In  Figure  10  show  more  complex  patterns,  with  more 
extensive  rechecks  during  assembly.  Visual  Inspection  of  some  profiles  suggests 
that  frames  4-5  and  9-11  are  provoking  search  strategies,  sometimes  requiring 
sampling  or  coordination  of  several  Information  searches. 

With  a  little  analytic  license.  It  Is  possible  to  reconstruct  the  Inform¬ 
ation  search  procedures  each  Individual  displayed.  Subject  3,  for  example, 
requested  a  moderate  number  of  defintlons  and  segmented  graphics  (but  not  over¬ 
view  graphics).  Faced  with  a  problem  at  frame  3,  4,  and  5,  though,  he  seemed 
to  rely  upon  rechecks  of  text.  At  frame  4,  this  Individual  requested  a  related 
segmented  graphic,  rechecked  frame  3,  then  moved  ahead  through  frame  4  to  frame  5. 
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Hert  he  requested  the  related  segmented  graphic,  rechecked  frame  4,  then 
■oved  through  frames  5,  6,  and  7.  At  this  point,  though,  feeling  unsure, 
he  once  again  returned  to  frame  4,  but  then  moved  forward  steadily,  with  no 
no  further  Information  requests  to  frame  9.  From  this  point  through  frame  15 
he  made  one  or  two  requests  at  each  frame.  At  frame  16  he  requested  definitions 
of  "Insert"  and  "grooves",  and  a  repeat  of  frame  15,  then  moved  to  frame  17, 
completing  the  assembly  task.  Following  frame  17  he  did  sweep  back  to  explore 
the  earlier  trouble  spot.  (This  exploration  Is  not  Included  In  the  analyses 
in  this  report.) 

Subject  5  used,  during  the  assembly  phase,  8  requests  for  deflations., 

7  rechecks  to  text  plus  1  for  graphics,  and  5  requests  for  graphics.  (There 
were  other  requests  during  the  exploratory  phase.)  In  trouble  spots  he  tended  - 
to  use  a  multi -media  search,  a  variety  of  Information  search  options.  At 
frame  4,  for  example,  this  individual  rechecked  to  frame  3,  moved  ahead  through 
frame  4  to  frame  5,  rechecked  to  frame  4,  viewing  the  related  graphic,  moved 
ahead  to  frame  5,  back  to  the  text  In  frame  4,  ahead  to  frame  5  to  check  twice 
the  definition  of  the  term  "angle  block",  back  to  frame  4,  ahead  to  frame  5 
for  the  same  definition.  From  here  he  returned  to  frame  5,  checking  the 
definition  a  third  time,  requested  the  accompanying  graphic  and  moved  on 
steadily  through  the  next  few  frames  without  any  Information  requests.  At 
frame  10  he  engaged  In  a  less  extended  search,  but  again  used  several  kinds 
of  Information.  After  finishing  the  asseiid)1y  task  at  frame  16  this  Individual 
returned  to  explore  several  points,  requesting  several  graphics  and  one  defini¬ 
tion  that  had  not  been  requested  during  the  assembly  phase. 


17 


These  "snapshots”  provide  an  Intriguing  view  of  the  range  of  strategies 
subjects  can  employ  In  mastering  directions  for  an  assembly  task,  given  a 
variety  of  kinds  of  Infbrmatlon  available  upon  request.  He'd  Ideally  like  to 
have  an  "x-ray"  of  the  covert  processes,  yet  even  In  these  less-than-perfect 
reconstructions,  combined  with  observations  and  with  comments  from  or  Infor¬ 
mal  debriefings  with  some  subjects,  we  can  see  some  patterns. 

Discussion 

This  report  Is  part  of  an  ongoing  effort  to  understand  the  processes  people 
employ  In  reading  technical  material  and  the  ways  In  which  Information  engineer¬ 
ing  can  facilitate  those  processes.  This  study  provided  a  detailed  and 
hierarchically  organized  Information  structure  (Hypertext)  as  part  of  a  computer 
assisted  job-aiding  procedure  for  an  assembly  task.  The  central  questions  were 
"Is  performance  (with  Hypertext  available)  greater  than  In  previous  studies?" 
"Uhen  people  have  easy  access  to  many  kinds  of  Information,  what  Information 
do  they  select  to  help  them  do  the  job?"  "How  do  program  sites  (Information 
blocks)  differ?"  "HOw  do  people  vary?"  and  "How  are  problem  solving  strategies 
displayed  during  computer  based  job  aiding?" 

Performance  on  the  Assembly  Task 

Performance  was  noticeably  better  than  In  previous  studies  employing  the 
same  task.  Crandell  &  Clock  (1981)  for  example,  presented  this  task  with  text 
and/or  graphics  on  separate  slide  projectors.  They  found  that  the  percentage 
of  subjects  (community  college  students)  with  errors  uncorrected  by  the  end 
were  1001  for  those  who  viewed  text  alone,  91 X  for  those  who  viewed  graphics 
alone,  and  75X  for  those  who  viewed  both.  In  the  present  study,  employing 
the  same  text  and  graphics  (though  graphics  In  their  study  were  "busier"  than 
those  In  this  study)  plus  definitions  In  a  rapid  access,  user  controlled  and 


computer  based  Information  structure,  only  one  subject  (5X)  who  had  an 
uncorrected  error  —  one  flat  plate  In  the  back  was  In  Its  proper  place  but 
backward. 

This  suggests  that  when  people  can  get  the  Information  they  need  when 
they  need  It,  performance  on  procedural  tasks  Is  facilitated.  But  since 
people  vary  in  the  Information  they  feel  they  need.  It  may  be  useful  to  have 
more  Information  (and  more  forms  of  Information)  available  than  any  one  person 
Is  likely  to  need,  yet  to  keep  the  surface  text  uncluttered  In  order  to  avoid 
Information  overload  for  those  who  don't  need  that  Information. 

This  study,  however,  did  not  directly  compare  performance  with  and  without 
hypertext.  At  this  point  we  could  not  rule  out  the  possibility  that  the  enhanced 
performance  was  due  simply  to  difference  In  subjects,  though  the  subjects  In  this 
study  ranged  widely  In  background;  some  were  considerably  less  familiar  with 
technical  material  than  subjects  In  earlier  studies  and  some  were  not.  A 
recently  completed  study  using  this  procedure  with  coimnunity  college  students 
should  help  to  clarify  this  point. 

The  study  was  on  one  level  the  application  of  a  new  variant  of  a  Job 
aiding  procedure  for  a  procedural  task.  On  another  level,  though,  the  breadth 
of  Information  available  to  subjects,  the  fact  that  subjects  could  select  the 
Information  they  felt  they  needed,  and  the  opportunity  to  monitor  closely 
subjects'  Information  selection  at  each  frame,  made  It  possible  to  describe 
overall  Information  selection  patterns  and  to  examine  variability  across 
subjects  and  across  program  frames.  This  In  turn  made  It  possible  to  explore 
the  applicability  of  the  model  of  Information  search  In  response  to  Information 
blocks,  embedded  within  the  broader  framework  on  on-line  problem  solving. 


Kinds  of  Information  Selection 

Subjects  as  a  group  selected  graphic  Information  most  often,  dictionary 
next,  and  sometimes  made  rechecks  of  text  or  nontext  Information.  Of  the 
types  of  graphics  available,  the  most  frequent  requests  were  for 
segemented  graphics  depicting  the  object  as  It  should  appear  at  a  given  point 
In  construction.  The  next  most  frequent  requests  were  for  definitions.  A 
few  dictionary  Items  were  requested  by  a  substantial  proportion  of  subjects, 
while  other  Items  were  requested  by  only  one  or  two  subjects.  Rechecks  to 
previous  texts,  not  used  by  all  subjects,  were  sometimes  brief,  returning 
to  the  previous  frame. 

Differences  Across  Program  Sites 

Responses  to  sites  In  the  text  differed  widely.  The  sixteen  program 
steps  were  In  effect  sixteen  different  problems  to  be  solved.  Some  were 
more  difficult  than  others,  and  some  were  difficult  for  different  reasons 
than  were  others.  Overall  difficulty  Is  reflected  In  the  total  number  of 
Information  requests  at  each  frame.  The  number  of  Information  requests  per 
frame  ranged  from  2  to  65.  Information  requests  were  most  common  for  frames 
4-5,  10-11,  and  frame  16. 

The  structural  difficulty  and  Invisibility  of  two  of  these  segments  may 
be  reflected  In  the  fact  that  requests  for  graphic  blowups  were  greatest  at 
frames  5  and  10.  At  frame  16,  though,  far  the  largest  proportion  of  requests 
was  for  definitions  of  one  or  two  terms  whose  meanings  or  referents  were 
unclear.  Some  frames  were  the  starting  point  for  far  more  rechecks  and 
Information  requests  for  Hypertext  than  were  others.  The  distribution  of 
Information  requests  may  be  useful,  as  discussed  later.  In  on-line  debugging 
procedures,  making  software  more  effective  and  less  costly. 


We  have  at  this  tine  only  a  partial  picture  of  the  link  between  type  of 
Information  block  and  type  of  Information  search,  but  some  tentative  generali¬ 
zations  nay  be  advanced.  Three  major  forces  seem  to  determine  the  number  and 
.  kinds  of  Information  searches  made  by  a  given  subject  at  a  given  Information 
block  In  this  assembly  task.  One  force.  Influencing  total  number  of  requests 
at  a  given  point.  Is  the  overall  difficulty  of  a  given  step.  Steps  4  and  5, 

10  and  11,  and  16  on  the  whole  provoked  a  relatively  large  number  of  requests, 
while  steps  2  and  15,  for  example,  provided  few. 

A  second  force  Influencing  both  number  and  type  of  requests  Is  Individual 
preference.  This  Is  based  on  prior  knowledge  (Influencing  subjective  dif¬ 
ficulty),  general  willingness  to  seek  Information  and  preference  for  particular 
kinds  of  Information.  One  Individual,  for  example,  made  no  requests  at  all, 
while  another  made  almost  50.  One  relied  primarily  on  graphics  and  much  less 
on  verbal  Information,  while  another  made  33  requests  for  verbal  Information 
and  only  3  for  graphics. 

A  third  force  seems  to  Involve  task  demands.  Steps  In  this  assembly  pro¬ 
cedure  differ  In  the  kinds  of  problems  they  present.  Steps  4  and  5,  for  example, 
may  be  difficult  primarily  because  they  Involve  the  Joining  of  two  subassemblies, 
for  which  a  spatial-graphic  overview  Is  likely  to  be  especially  useful,  though 
some  need  for  definitions  Is  also  present.  Steps  9  and  10  provoked  requests 
both  for  graphics  and  for  definitions.  These  steps  Involve  the  Joining  of  two 
subassemblies,  described  expllcitlly  but  not  necessarily  clearly  In  the  text. 

7V(d  other  problem  sources  may  be  Involved  here  —  Invisibility  and  ambiguity. 

The  Joint  Is  not  entirely  visible  from  the  segemented  graphic  or  overview 
graphic,  and  some  of  the  terms  used  seem  easy  enough,  but  subjects  may  not 
be  sure  what  they  refer  to. 
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The  general  approach  In  this  research  program  has  been  to  observe  what 

happens  In  a  moderate-fidelity  simulation,  rather  than  to  manipulate  the 

dimensions  of  that  simulation.  The  text  and  graphic  Information  were  designed 
to  help.  The  Information  blocks  that  remained,  though  not  deliberately  planned 

or  manipulated,  were  not  excessive— many  technical  manuals.  In  fact,  seem  to 
have  been  designed  to  guard  against  comprehension.  At  some  point,  though.  It 
would  be  desirable  and  feasible  to  manipulate  some  features  of  the  program  to 
assess  effects  of  various  kinds  of  Information  blocks.  For  example.  It  might 
be  possible  to  reduce  vocabulary  difficulty  at  frames  10-11  by  defining  terms 
In  context  or  In  rebus,  and  to  analyze  whether  that  change  reduced  overall 
difficulty  and  requests  for  dictionary  but  had  little  effect  on  requests  for 
graphics. 

Or  It  might  be  possible  to  alleviate  structural  difficulty  either  by 
Inserting  In  the  surface  text  a  minimal  schematic  showing  the  relationship 
of  the  key  subassemblies  or  by  "framing"  with  text.  For  example,  we  could 


add  to  the  text  "Earlier  you  made  the  back  and  added  an  axle.  Now  you  will 
Join  the  base  to  the  axle  at  a  right  angle."  Again,  It  would  be  possible  to 
examine  whether  such  changes  affected  not  only  overall  difficulty  but  the 

kinds  of  Information  requested.  _ 

Not  only  can  problems  be  reduced,  they  can  deliberately  be  Increased 
In  order  to  understand  their  Impact.  To  assess  the  effect  of  the  segmented 


graphics  (or  rechecks  or  definitions),  for  example,  that  feature  could  be 
locked  out.  Or,  to  Induce  search,  subjects  might  be  directed  to  join  two 
components  that  cannot  at  that  point  be  joined.  Over  the  long  run  It  would 
obviously  be  useful  to  develop  new  tasks,  but  there  are  also  advantages  In 
systematic  variations  on  well-mapped  tasks,  manipulating  experimentally 
factors  suggested  by  more  naturalistic  Investigations  of  real  or  simulated 


tasks. 


vr. ' 


In  ordtr  to  understand  the  relationship  of  user,  task,  and  technical  1n- 
forMtion  systen,  it  will  in  the  long  run  be  necessary  to  categorize  more 
precisely  the  types  of  the  information  blocks  encountered.  This  will  not  be 
easy.  As  indicated  earlier,  one  tactic  that  can  be  used  in  this  effort  is  to 
-construct  and  systematically  vary  types  of  information  blocks.  Even  then, 
though,  problems  will  remain.  One  is  that  an  information  block  is  to  a  certain 
extent  an  individual  matter;  one  person  may  be  unfazed  while  another  is  stopped. 
Yet  by  studying  both  group  trends  and  individual  responses  we  may  be  able  to 
come  to  clearer  understanding  of  the  most  common  types  of  Information  blocks 
and  the  types  of  information  that  tend  to  resolve  them.  Similarly,  each 
information  block  is  to  some  extent  situation-specific,  conditioned  by  other 
features  of  the  text  and  graphic  environment.  Yet  we  can  abstract  the  general - 
izable  factors  from  a  variety  of  specific  Instances. 

A  third  problem  is  that  information  is  diffuse.  If  information  is  not 
available  at  one  place  It  may  be  stated  or  inferrable  at  another  place  or  in 
another  format  or  in  another  modality  (making  the  task  easier  for  the  user 
and  harder  for  the  researcher).  Type  and  criticality  of  information  block 
in  a  subtask  may  Interact  with  availability  of  alternate  sources  in  the  infor¬ 
mation  system  and  with  user  characteristics  such  as  motivation,  preference 
for  various  forms  of  information,  and  skill  in  constructing  inferential 


networks. 


Di fferences  across  Subjects 


Subjects  differed  in  the  number  of  information  requests  they  made.  The  total 


number  of  requests  per  subject  ranged  from  o  to  46,  with  a  mean  of  20.  The  number 
of  requests  for  graphics  per  subject  ranged  from  0  to  27,  with  a  mean  of  11. 


The  number  of  requests  for  verbal  Information  per  subject  ranged  from  0  to  33,  with 
a  mean  of  10.  Subjects  also  varied  In  the  patterning  of  their  Information  requests. 
Some  made  few  or  no  requests  for  Information  to  supplement  the  text.  Others,  faced 
with  an  Information  block,  quickly  sought  and  received  Information  from  a  single 
source.  Others,  faced  with  the  same  block,  engaged  In  a  multi-media  search,  a 
concatenation  of  graphics,  defintlons,  and  rechecks  to  previous  text,  all  brought 
to  bear  on  the  problem.  k 

The  task  In  this  study  was  straightforward, objective,  constrained  not  only 
by  the  very  directive  directions,  but  by  the  very  concrete  materials— they  could 
be  assembled  In  only  a  limited  number  of  ways,  and  yet  there  were  times  we  wondered 
whether  this  was  a  projective  test,  a  Rorschach  In  on-line  graphics.  At  least  some 
subjects  seemed  to  use  this  task  In  a  larger,  self-defined  task,  acting  out  and 
testing  some  statements  about  themselves.  One  subject  reported  that  she  was  mad 
at  herself  when  she  had  to  give  In  and  ask  for  help.  Similarly,  another,  also  asking 
for  Information  only  when  It  was  essential,  and  only  after  she'd  completed  a  step, 
laughed  when  she  recognized  her  own  competitivness.  Two  other  subjects  mentioned 
they  had  been  very  careful  not  to  make  any  mistakes,  e  goal  not  stressed  by  the 
researchers.  Though  no  attempt  was  made  to  quantify  these  personal  meanings.  It 
Is  possible  that  these  subjective  Interpretations  and  more  objective  metacognitions 
about  when  to  stop  seeking  Information  and  what  sources  are  likely  to  be  useful 
combine  with  learning  strategy  preference  to  form  a  problem  space  that  the  indivl- 
ual  superimposes  on  the  problem  as  defined  by  the  Investigator.  It  might  be  pos¬ 
sible  to  induce  greater  conformity  In  approach,  perhaps  by  emphasizing  more  strongly 
that  the  objective  was  to  complete  assembly  of  the  loading  cart  with  no  errors,  but 
that  they  could  have  as  much  time  and  as  much  Information  as  they  wanted.  Yet  at 
this  stage  we  wanted  to  learn  as  much  as  we  could  about  how  people  thought  about  the 


To  retrace  the  steps  In  problem-solving  described  earlier.  In  the  Input 
translation  phase  of  the  overall  task  an  Individual  seems  In  general  to  accept 
the  goal  as  defined  In  the  program  and  the  resources  provided.  For  those  with 
any  experience  along  this  line,  the  general  nature  of  the  assembly  task  Is 
readily  apparent,  and  relevant  strategies  are  available.  He  or  she  may,  however, 
fall  to  remember  all  of  the  potential  Information  sources  and  may  add  constraints 
of  his/her  own,  such  as  using  as  little  Information  as  possible  or  avoiding  any 
errors  or  rechecks. 

Selection  and  application  of  a  problem  solving  method,  constrained  to 

some  degree  by  the  Input  translation  of  the  Information  potential  of  the  text, 
Hypertext,  and  concrete  materials  (and  explicit  directions)  proceeds  In  this 
context  by  use  of  what  Is  In  sight  or  In  memory,  and  by  explicit  selection  of 
additional  Information  from  the  computer-based  technical  Information  system. 

In  this  situation.  It  seems  useful  to  treat  together  selection  and  application, 
although  some  subjects,  because  of  their  previous  experience,  may  have  been 
somewhat  more  Inclined  to  select  than  able  to  apply  Information  such  as  graphics. 
On  the  whole,  though,  these  were  able  subjects  and,  on  the  molecular  level,  able 
to  apply  the  Information  they  selected.  On  a  more  molar  level  the  method  In¬ 
cludes  the  subject's  evaluation  of  the  kind  of  Information  block,  the  Information 
move(s)  most  likely  to  resolve  It,  and  self-monitoring  of  the  success  of  those 
efforts.  On  this  level,  not  examined  here,  or  even  on  the  molecular  level  for 
less  able  subjects.  It  may  be  useful  to  separate  selection  and  application 
and  even  to  break  down  these  steps  further. 

Regrouping  may  be  seen  In  the  application  of  varied  Information  seeking 
methods  to  stubborn  trouble  spots,  and  hinted  In  the  occasional  sweeps  back 
to  an  earlier  frame.  We  have  no  direct  record  of  any  changes  In  a  subject's 


problem  representation.  There  Mas  no  Indication  that  subjects  abandoned  the 
attempt  to  solve  any  of  the  problems  they  encountered.  Yet  In  a  task  such  as 
this,  an  Individual  may  be  less  than  perfectly  certain  about  one  of  the  steps 
and  may  at  some  point  pass  on,  even  If  some  degree  of  uncertainty  remains. 

The  generation  of  subgoal s  was  not  often  seen(and  would  be  hard  to  document 
if  set  aside).  Within  some  complex  concatenations  of  strategies,  though, 
such  a  step  In  problem  solving  Is  suggested.  One  subject,  for  example.  In 
attempting  to  solve  the  problem  encountered  at  frame  5,  rechecketf  to  frames 
3  and  4,  recognizing  after  passing  these  points  that  there  was  need  to  dis¬ 
ambiguate  some  terms  or  spatial  orientations  before  the  central  goal  of  frame 
5  could  be  achieved.  Some  of  the  sweeps  backward  during  the  exploration  phase 
may  reflect  a  return  to  subgoals  set  during  the  assembly  task.  A  few  subjects 
seemed  either  to  pass  a  step  In  assembly  with  some  unanswered  questions  or  with 
a  question  about  why  they  had  at  first  failed  to  understand  a  point,  and  returned 
during  the  optional  exploration  period  to  reexamine  the  subgoal  (of  subje.w(ve 
sense  of  clear  understanding)  earlier  set  aside. 

Experts  on  problem  solving  such  as  Newell  and  Simon  (1972)  or  Shulman  and 
El  stein  (1975)  Invested  many  years,  close  analysis,  and  much  thoughtful  analysis 
and,  undoubtedly,  reconceptualization  before  presenting  formal  models  of  the 
process.  We  would  not  be  so  bold  as  to  present  a  formal  model  at  this  point. 

The  results  we  present  here  are  only  a  beginning,  and  we  have  elected  to  start 
from  the  outside  with  objectively  observable  behavior,  then  gradually  to  circle 
closer. 

Impl Icatlons 

Implications  for  Practice 

Where  users  differ  In  preference  and  ability  to  process  Information  In 
different  formats  and  different  levels  of  generality.  It  may  be  useful  to 
Include  a  variety  of  kinds  of  Information  In  a  technical  Information  system 
for  job  aiding,  and  to  allow  subjects  a  high  degree  of  control  over  Information. 
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This  Is  suggested  by  the  Increase  In  performance  over  previous  studies  and 
by  the  Individual  differences  In  number  and  kinds  of  Information  requested. 

It  will,  however,  be  Important  to  test  these  factors  more  systematically  than 
was  attempted  In  this  study. 

Other  Implications  for  practice  lie  In  combination  of  an  Internal  tracking 
system  and  response  analysis  system,  which  can  be  used  to  develop  Individual 
response  profiles,  perhaps  mapped  against  Ideal  or  typical  profiles,  providing 
on-line  Diagnostics  of  user's  response  patterns.  These  In  turn  can  be  linked 

to  Adviser  messages.  Intervention  In  the  form  of  key  Information  If  the  user 
seems  lost,  or  signals  to  the  trainer  or  supervisor.  Cumulation  of  Information 
selection  patterns  In  a  Debug  subroutine  across  a  relatively  small  number  of 
subjects  can  help  to  spot  potential  problems  and  suggest  remedies  before 
courseware  leaves  the  shop,  reducing  sharply  the  long  range  (Bunyard  & 

Coward,  1982)  costs. 

Implications  for  Systematizing  Knowledge  about  Job  Aiding 

Job  aiding  has  drawn  from  laboratory  research.  Instructional  design  folklore, 
and  the  designer's  Intuition,  That's  not  good  enough.  In  a  period  of  rapidly 
changing  technology  many  workers  will  be  left  out  and  many  complex  systems 
endangered  unless  job  aiding  systems,  often  computer-based,  provide  effective 
assistance.  The  knowledge  currently  available  should  be  more  systematically 
organized  and  the  potent  variables  carefully  tested  In  work-like  tasks. 

The  job  aiding  procedure  described  In  this  study  has  promise  as  a  base  for 
assessing  quite  directly  the  effects  of  single  variables  and  complex  com¬ 
binations.  For  example,  It  would  be  possible  to  test  the  effects  of  text  only 
vs.  segmented  graphics  only  vs.  overview  graphics  only,  with 
and  without  the  recheck  option.  Or  to  test  a  complex  Individually  controlled 


system  like  the  one  used  here  against  an  equally  complex  system  controlled  by 
the  experimenter/ trainer  or  program,  based  on  response  analysis  or  characteris¬ 
tics  such  as  user's  level  of  expertise  or  learning  strategy  preference. 


The  Implications  fbr  construction  of  a  theoretical  base  for  design  of  Job 
aiding  are  not  In  the  results  of  this  small  study  but  In  the  possibilities  It 
raises  for  alternation  of  Integration  of  literature,  synthesis,  tentative  model 
building,  direct  applications,  and  empirical  testing  of  practically  and  theore¬ 
tically  Inqrartant  variables.  The  strongest  argument  for  this  approach  Is  that 
It  stays  close  to  real  problems  and  real  people,  reducing  the  ultimate  difficulty 
of  translating  theory  Into  practice.  And  while  research  on  text  processing  has 
contributed  to  computer  based  Job  aiding,  research  on  Job  aiding  can  contribute  to 
research  and  practice  In  the  broader  field  of  technical  literacy  (Hutson,  1982). 
Implications  for  Studying  On-Line  Problem  Solving 

This  study  has  answered  a  few  obvious  questions,  formulated  a  few  less- 
obvious  questions,  and  provided  a  simple  conceptualization  of  Information 
blocks  and  Information-  search  strategies  as  well  as  procedures  for  gathering, 
representing  and  analyzing  relevant  data.  It  has  attempted  to  fit  but  not 
force  these  phenomena  Into  a  classic  model  of  problem  solving,  with  special 
emphasis  on  the  Information  selection  processes. 

The  TICCIT  system's  Internal  tracking  system  made  It  possible  to  record 
for  each  subject  at  each  step  the  exact  sequence  of  information  selection, 
allowing  some  Inferences  about  thought  processes,  based  on  subjects' 
profiles,  observer's  notes,  and  Informal  debriefing.  One  future  direction  Is 
to  obtain  from  some  subjects  a  verbal  protocol  as  they  work.  While  It's 
possible  that  the  verbal  protocol  would  change  the  task,  the  objective  record 

of  Information  moves  could  be  compared  to  profiles  obtained  without  the 
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th1nk<-a1oud  feature.  The  verbal  protocols  should  be  especially  helpful  In 
determining  users'  problem  representations  and  metacognitions,  enriching 
our  interpretations  of  the  more  easily  quantified  machine-recorded  responses. 

Another  possible  direction  Is  to  provoke  awareness  of  search  processes 
by  violating  expectations.  Just  es  social  conventions  are  most  likely  to  be 
made  explicit  by  the  response  when  they  are  violated,  thought  processes  may  be 
most  sharply  defined  when  a  subject  finds  his/her  expectations  violated.  In 
the  present  study,  the  Information  In  text  and  graphics  was  congruent  and 
complementary,  and  the  sequence  reasonable.  But  It  would  be  easy  to  Insert  a 
few  points  where  there  was  a  contradiction  or  apparent  contradiction  between 
text  and  graphics  or  between  two  lines  In  the  text.  Another  possibility  would 
be  to  violate  constraints  on  sequence,  e.g.,  directing  subjects  to  unite 
one  sub-assembly  to  another  that  has  not  yet  been  assembled.  In  resolving 
these  problems,  subjects  might  make  more  visible  their  thought  processes. 

From  this  point  there  are  a  number  of  different  paths  that  could  be 
explored  In  studying  on-line  search  processes,  and  we  hope  that  others  will 
Join  us  In  this  exploration.  . 
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Table  2 

Frequency  of  Various  Types  of  Information  Searches 
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Figurt  1.  Exanples  of  steps  In  surface  text 
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Figure  4.  A  schematic  representation  of  the  Information  search 
strategies  for  resolving  an  Information  block  when  Hypertext  display 
Is  available.  Options  Include  moving  forward,  moving  up  to  a  graphic 
overview,  down  to  clarification  of  a  specific  term,  or  rechecking  the 
surface  text  or  Hypertext. 
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Personnel  Technology 
Office  of  the  Dnder  Secretary  of  Defens 
for  Keseareh  &  Engineering 
Boob  3D129,  The  Pentagon 
Washington,  DC  20301 

1  Major  Jack  Thorpe 
DARPA 

1400  Wilson  Blvd. 

Arlington.  VA  22209 

1  Dr.  Robert  A.  Wisher 
ODSDRE  (ELS) 

The  Pentagon,  Room  3D129 
Washington.  DC  20301 
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Civilian  Agencies 

1  Dr.  Patricia  A.  Butler 
MIC-BRN  Bldg.  Stop  #  7 
1200  19th  St..  MW 
Washington.  DC  20208 

1  Dr.  Joseph  L.  Toung.  Director 
Menory  6  Cognitive  Froceases 
National  Science  Foundation 
Washington.  DC  20550 


Private  Sector 

1  Dr.  John  R.  Anderson 
Departuent  of  Psychology 
Carnegle-Mellon  University 
Pittsburgh.  PA  15213 

1  Dr.  John  Black 
Yale  University 
Box  llA,  Yale  Station 
Mew  Raven,  CT  06520 

1  Dr.  John  S.  Brown 
XEROX  Palo  Alto  Research  Center 
3333  Coyote  Road 
Palo  Alto,  CA  94304 

1  Dr.  Glenn  Bryan 
6208  Poe  Road 
Bethesda,  HD  20817 

1  Dr.  Pat  Carpenter 
Department  of  Psychology 
Carnegle-Mellon  University 
Pittsburgh,  PA  15213 

1  Dr.  Mlcheline  Chi 
Learning  RAD  Center 
University  of  Pittsburgh 
3939  O'Hara  Street 
Pittsburgh,  PA  15213 

1  Dr.  William  Clancey 
Department  of  Computer  Science 
Stanford  University 
Stanford,  CA  94306 

1  Dr.  Michael  Cole 
University  of  California 
at  San  Diego 

Laboratory  of  Comparative 
Human  Cognition  -  D003A 
U  Jolla.  CA  92093 

1  Dr.  Allan  H.  Collins 
Bolt  Beranek  A  Newman,  Inc. 

50  Moulton  Street 
Cambridge,  MA  02138 


1  Dr.  Thomas  M.  Duffy 
Department  of  English 
Carnegle-Mellon  University 
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Private  Sector 

1  me  Pacllltf-Acqulaltlona 
4833  Kugby  Avenue 
■etbeada.  MD  20014 

1  Dr.  Aodera  Crleaaon 
Departaent  of  Peychology 
Oniveralty  of  Colorado 
Boulder,  CO  80309 

1  Mr.  Vallaec  Feurreig 
Departaent  of  Educational  Technology 
Bolt  Beranek  k  Ncuaan 
10  Moulton  St. 

Caabrldge,  MA  02238 

1  Dr.  John  B.  Frederlkaen 
Bolt  Beranek  k  Neuaan 
50  Moulton  Street 
Caabrldge,  KA  02138 

1  Dr.  Dedre  Centner 
Bolt  Beranek  k  Nevaan 
10  Moulton  St. 

Caabrldge,  HA  02138 

1  Dr.  lobert  Claaer 

Learning  Beaearcb  k  Developaent  Center 
Dnlveralty  of  Flttaburgh 
3939  O'Hara  Street 
PITTSBORGH.  PA  15260 

1  Dr.  Joaph  Coguen 
SBl  International 
333  Bavenauood  Avenue 
Menlo  Park.  CA  94025 

1  Dr.  Daniel  Gopher 
Faculty  of  Induatrlal  Engineering 
k  Managcaent 
TECHNION 
Haifa  32000 
ISRAEL 


1  DR.  JAMES  C.  CREERO 

uoc 

ONIVERSm  OF  PITTSBURGH 
3939  O'HARA  STREET 
PITTSBURGH.  PA  15213 

1  Dr.  David  Kleraa 
Departaent  of  Paycbology 
Dnlveralty  of  Arltona 
Tuaeon,  AZ  85721 


Private  Sector 

1  Dr.  Halter' Klntach 
Departaent  of  Paychology 
Dnlveralty  of  Colorado 
Boulder.  CO  80302 

1  Dr.  Stephen  Koaalyn 
1236  Ullllaa  Jaaea  Hall 
33  Kirkland  St. 

Caabrldge,  NA  02138 

1  Dr.  Jill  Larkin 
Departaent  of  Paychology 
Carnegie  Mellon  Unlveralty 
Flttaburgh,  PA  15213 

1  Dr.  Alan  Leagold 
Learning  RkD  Center 
Dnlveralty  of  Plttaburgh 
3939  O'Hara  Street 
Plttaburgh,  PA  15260 

1  Dr.  Jla  Levin 
Unlveralty  of  California 
at  San  Diego 

Laboratory  tot  Coaparatlve 
Hunan  Cognition  -  D003A 
La  Jolla.  CA  92093 

I  Dr.  Don  Lyon 
AFHRL/OT  (UDRl) 

Hllllaaa  AFB,  AZ  85225 

1  Dr.  Jaaea  R.  Miller 
Ceaputer*Thought  Corporation 
1721  Heat  Plano  Highway 
Plano,  TX  75075 

1  Dr.  Ton  Horan 
Xerox  PARC 

3333  Coyote  Hill  Road 
Palo  Alto.  CA  94304 

1  Dr.  Allen  Munro 

Behavioral  Technology  Laboratorlea 
1845  Elena  Ave.,  Fourth  Floor 
Redondo  Beach.  CA  90277 

1  Dr.  Donald  A  Homan 
Cognitive  Science.  C-015 
Dnlv,  of  California,  San  Diego 
U  Jolla.  CA  92093 
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Private  Sector 

1  Dr.  Jeaae  Orlanaky 
Institute  for  Defense  Analyses 
1801  H.  Beauregard  St. 

Alexandria.  VA  22311 

1  Dr.  Haney  Pennington 
University  of  Chicago 
Graduate  School  of  Business 
1101  E.  S8th  St. 

Chicago.  IL  60637 

1  DR.  PETER  POLSON 
DEPT.  OP  PSYCHOLOGY 
DNlVERSm  OP  COLORADO 
BOULDER.  CO  80309 

1  Dr.  Fred  Relf 
Physics  Departtaent 
University  of  California 
Berkeley.  CA  94720 

1  Dr.  Lauren  Resnlck 
IROC 

University  of  Pittsburgh 
3939  O'Hara  Street 
Pittsburgh.  PA  1521 

1  Mary  S.  Riley 
Progran  in  Cognitive  Science 
Center  for  Hunan  Infomatlon  Processing 
University  of  California,  San  Diego 
La  Jolla.  CA  92093 

1  Or.  Andrew  N.  Rose 
Anerlcan  Institutes  for  Research 
1055  Thonas  Jefferson  St.  HH 
Washington,  DC  20007 

1  Dr.  Emat  Z.  Rothkopf 
Bell  Laboratories 
Murray  Hill.  HJ  07974 

1  Dr.  Hllliaa  B.  Rouse 
Georgia  Institute  of  Technology 
School  of  Industrial  6  Syatens  • 
Engineering 
Atlanta,  CA  30332 

1  Dr.  Halter  Schneider 
Psychology  Dspartnent 
603  E.  Daniel 


Private  Sector 

1  Dr.  H.  Wallace  Sinalko 
Progran  Director 

Manpower  Research  and  Advisory  Services 
Snlthaonlan  Institution 
801  North  Pitt  Street 
Alexandria,  VA  22314 

1  Dr.  Edward  E.  Snlth 
Bolt  Beranek  A  Newnan,  Inc. 

50  Moulton  Street 
Canbrldge.  MA  02138 

1  Dr.  Eliott  Soloway 
Yale  University 
Departnent  of  Conputer  Science 
P.O.  Box  2158 
New  Haven.  CT  06520 

1  Dr.  Kathryn  T,  Spoehr 
Psychology  Departnent 
Brown  University 
Providence.  RI  02912 

1  Dr.  Robert  Sternberg 
Dept,  of  Psychology 
Yale  University 
Box  llA,  Yale  Station 
New  Haven.  CT  06520 

1  David  E.  Stone.  Ph.D. 

Hazeltlne  Corporation 
7680  Old  Sprlnghouse  Road 
MeUan,  VA  22102 

1  Dr.  Klkunl 'Tatsuoka 
Conputer  Based  Education  Research  Lab 
252  Engineering  Research  Laboratory 
Urbans.  IL  61801 

1  Dr,  Perry  H.  Thorndyke 
Perceptronlcs,  Inc. 

545  Mlddlefleld  Road.  Suite  |40 
Menlo  Park.  CA  94025 

1  Dr.  Kurt  Van  Lehn 
Xerox  PARC 

3333  Coyote  Hill  Road 
Palo  Alto.  CA  94304 


